---
id: artillery-plugin
title: Performance and Trace-Based Tests with Tracetest and Artillery Plugin
description: Quickstart on how to use the Tracetest x Artillery Integration to enhance Performance Tests with Trace-Based Testing using Tracetest.
hide_table_of_contents: false
keywords:
  - tracetest
  - trace-based testing
  - observability
  - distributed tracing
  - testing
  - artillery
  - load testing
  - performance testing
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---

:::tip Commercial Feature
[Feature available only in Cloud-based Managed Tracetest & Enterprise Self-hosted Tracetest.](https://tracetest.io/pricing)
:::

:::info Version Compatibility
This integration is compatible with [Artillery v2.0.10](https://github.com/artilleryio/artillery/releases/tag/artillery-2.0.10) and above.
:::

:::note
[Check out the source code on GitHub here.](https://github.com/kubeshop/tracetest/tree/main/examples/quick-start-artillery)
:::

[Tracetest](https://app.tracetest.io/) is a testing tool based on [OpenTelemetry](https://opentelemetry.io/) that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

[Artillery](https://artillery.io/) is a modern, powerful load-testing toolkit. Artillery is designed to help developers and testers simulate traffic to their applications, APIs, and microservices. It allows users to define scenarios to test how their systems behave under different loads.

## Why is this important?

Artillery is it's a great tool in its own right that allows you to replicate most of the production challenges you might encounter. But, as with all of the tools that only test the initial transaction between the client side and the server, you can only run validations against the immediate response from the service.

## The `artillery-plugin-tracetest` NPM Package

The [`artillery-plugin-tracetest`](https://www.npmjs.com/package/artillery-plugin-tracetest) package allows you to extend the Artillery's HTTP engine to enhance your performance tests with trace-based testing. It uses Artillery's plugin system to intercept requests and attaches the trace ID to the request headers. This allows you to run trace-based tests using the Tracetest API.

:::note
Do you want to use Tracetest Triggers as an entry point for your performance tests? Take a look at the [`artillery-engine-tracetest`](./artillery-engine) for a complete guide on how to do it.
:::

## Today You'll Learn How to integrate Trace-Based Tests with your Aritllery Test Scripts

This is a simple quick-start guide on how to use the Tracetest `artillery-plugin-tracetest` NPM package to enhance your Artillery Test Scripts with trace-based testing. The infrastructure will use the Pokeshop Demo as a testing ground, triggering requests against it and generating telemetry data.

## Requirements

**Tracetest Account**:

- Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/overview) docs.
- Create an [environment](/concepts/environments).
- Create an [environment token](/concepts/environment-tokens).
- Have access to the environment's [agent API key](/configuration/agent).

**Docker**: Have [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your machine.

## Run This Quckstart Example

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the TypeScript Quickstart:

```bash
git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-artillery
```

Follow these instructions to run the included demo app and TypeScript example:

1. Copy the `.env.template` file to `.env`.
2. Log into the [Tracetest app](https://app.tracetest.io/).
3. This example is configured to use Jaeger. Ensure the environment you will be utilizing to run this example is also configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, updating the URL to `jaeger:16685`, Test Connection and Save.
4. Fill out the [token](https://docs.tracetest.io/concepts/environment-tokens) and [agent API key](https://docs.tracetest.io/concepts/agent) details by editing your `.env` file. You can find these values in the Settings area for your environment.
5. Run `docker compose up -d`.
6. Run `npm i` to install the required dependencies.
7. Run `npm run test:plugin` to run the example.
8. The output will show the test results and the Tracetest URL for each test run.
9. Follow the links in the log to view the test runs programmatically created by the Atillery execution.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

## Project Structure

The quick-start Artillery project is built with Docker Compose.

The [Pokeshop Demo App](/live-examples/pokeshop/overview) is a complete example of a distributed application using different back-end and front-end services. We will be launching it and running tests against it as part of this example.

The `docker-compose.yaml` file in the root directory of the quick start runs the Pokeshop Demo app, the OpenTelemetry Collector, Jaeger, and the [Tracetest Agent](/concepts/agent) setup.

The Artillery Plugin quick start has two primary files: a Test Script file `plugin-test.yaml` that defines the Artillery execution, and a Tracetest Definition file `import-pokemon-trace-id.yaml` that contains the specs and execution of the trace-based tests.

## Installing the `artillery-plugin-tracetest` NPM Package

The first step when using the Artillery Plugin NPM package is to install the `artillery-plugin-tracetest` NPM Package. It is as easy as running the following command:

```bash
npm i artillery-plugin-tracetest
```

Once you have installed the `artillery-plugin-tracetest` package, you can use it as part of your Artillery Test Scripts to trigger trace-based tests and run checks against the resulting telemetry data.

## Tracetest Test Definitions

The `import-pokemon-trace-id.yaml` file contains the YAML version of the test definitions that will be used to run the tests. It uses the Artillery trigger to execute requests against the Pokeshop Demo.

```yaml
type: Test
spec:
  id: artillery-plugin-import-pokemon
  name: "Artillery Plugin: Import a Pokemon"
  trigger:
    type: traceid
    traceid:
      id: ${var:TRACE_ID}

  # test specs and assertions
  specs:
    - selector: span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"]
      name: "All HTTP Spans: Status  code is 200"
      assertions:
        - attr:http.status_code = 200
    - selector: span[tracetest.span.type="http" name="GET" http.method="GET"]
      assertions:
        - attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"
    - selector: span[tracetest.span.type="database"]
      name: "All Database Spans: Processing time is less than 1s"
      assertions:
        - attr:tracetest.span.duration < 1s
    - selector: span[tracetest.span.type="http" name="post" http.method="POST"]
      name: Response time should be fast
      assertions:
        - attr:response.time.ms < 100

  # generated output
  outputs:
    - name: DATABASE_POKEMON_ID
      selector: span[tracetest.span.type="database" name="create postgres.pokemon" db.system="postgres" db.name="postgres" db.user="postgres" db.operation="create" db.sql.table="pokemon"]
      value: attr:db.result | json_path '$.id'
```

## Creating the Artillery Test Script

The `plugin-test.yaml` file contains the Artillery Test Script that will be used to trigger requests against the Pokeshop Demo and run trace-based tests. The steps executed by this script are the following:

1. Congures a phase that will execute 10 requests against the Pokeshop Demo url `http://localhost:8081`.
2. Defines two plugins, the `public-metrics` to send Artillery's telmetry data to the OpenTelemetry Collector and the `tracetest` plugin.
3. Includes the API `token` to access the Tracetest APIs and the Tracetest `definition`.
4. Defines the scenarios to execute triggering a request to the `POST pokemon/import` endpoint sending `6` (Charizard) as the Pokemon Id.

```yaml
config:
  target: "http://localhost:8081"
  phases:
    - duration: 2
      arrivalRate: 5
  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: "artillery"
        metrics:
          reporter: otlp-http
          endpoint: "http://localhost:4318/v1/metrics"
          attributes:
            environment: "test"
            tool: "Artillery"
            type: "Load test"

        traces:
          reporter: otlp-http
          endpoint: "http://localhost:4318/v1/traces"
          attributes:
            environment: "test"
            tool: "Artillery"
    tracetest:
      token: tttoken_dfa1e954d47d1857
      definition: import-pokemon-trace-id.yaml
scenarios:
  - name: "Import Pokemon"
    flow:
      - post:
          url: "/pokemon/import"
          json:
            id: 6
```

## Running the Full Example

To start the full setup, run the following command:

```bash
docker compose up -d
npm run test:plugin
```

## Finding the Results

The output from the Tracetest Plugin script should be visible in the console log after running the test command. This log will show links to Tracetest for each of the test runs invoked by the Artillery Testing Script. Click a link to launch Tracetest and view the test result.

```bash
> quick-start-artillery@1.0.0 test:plugin
> artillery run plugin-test.yaml

Test run id: t6pjz_zbaqcap7rhab59xn9kygn66bg8cfq_fqjc
Phase started: unnamed (index: 0, duration: 2s) 15:59:46(-0600)

Phase completed: unnamed (index: 0, duration: 2s) 15:59:48(-0600)

--------------------------------------
Metrics for period to: 15:59:50(-0600) (width: 2.509s)
--------------------------------------

http.codes.200: ................................................................ 10
http.downloaded_bytes: ......................................................... 80
http.request_rate: ............................................................. 10/sec
http.requests: ................................................................. 10
http.response_time:
  min: ......................................................................... 1
  max: ......................................................................... 2
  mean: ........................................................................ 1.6
  median: ...................................................................... 2
  p95: ......................................................................... 2
  p99: ......................................................................... 2
http.responses: ................................................................ 10
tracetest.tests_started: ....................................................... 10
vusers.created: ................................................................ 10
vusers.created_by_name.Import Pokemon: ......................................... 10


--------------------------------------
Metrics for period to: 16:00:10(-0600) (width: 5.71s)
--------------------------------------

tracetest.tests_failed: ........................................................ 2
tracetest.tests_succeeded: ..................................................... 1
vusers.completed: .............................................................. 3
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 16656.5
  max: ......................................................................... 22462.5
  mean: ........................................................................ 18737.5
  median: ...................................................................... 17158.9
  p95: ......................................................................... 17158.9
  p99: ......................................................................... 17158.9


All VUs finished. Total time: 30 seconds

--------------------------------
Summary report @ 16:00:17(-0600)
--------------------------------

http.codes.200: ................................................................ 10
http.downloaded_bytes: ......................................................... 80
http.request_rate: ............................................................. 5/sec
http.requests: ................................................................. 10
http.response_time:
  min: ......................................................................... 1
  max: ......................................................................... 2
  mean: ........................................................................ 1.6
  median: ...................................................................... 2
  p95: ......................................................................... 2
  p99: ......................................................................... 2
http.responses: ................................................................ 10
tracetest.tests_failed: ........................................................ 9
tracetest.tests_started: ....................................................... 10
tracetest.tests_succeeded: ..................................................... 1
vusers.completed: .............................................................. 10
vusers.created: ................................................................ 10
vusers.created_by_name.Import Pokemon: ......................................... 10
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 16656.5
  max: ......................................................................... 27889.9
  mean: ........................................................................ 23730.5
  median: ...................................................................... 25091.6
  p95: ......................................................................... 27181.5
  p99: ......................................................................... 27181.5
⠸ ✔ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/104) - trace id: 787ff20c554c1e21cc90a7c3eec8d2ee

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/105) - trace id: 44a208bbe579c513c5f79ac8838aeb46
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/106) - trace id: a677b4c7b68f4d692087c860686699fc
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/107) - trace id: e6e3963c0975367d89308acd4e3c379c
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/108) - trace id: 22684fc7a5d43ed9e44e24c7135a6aa5
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/109) - trace id: c73da8f9a009dc365f19f8701924c9cc
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/110) - trace id: 98c74bd6e6501f6721f058be4f5bf311
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/111) - trace id: 5c722a53b527f7334800dd42d98f5a49
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/112) - trace id: 53ea977eb682fc5dc416f0425dc0ac5e
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/113) - trace id: d1de80e71c85119b62c8495f3ec42e81
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])
```

:::note
Most of the tests will fail as the import Pokemon flow reads from memory if the info already exists. This is expected behavior.
:::

:::tip View these tests in our Demo environment
[👉 Join our shared Pokeshop API Demo environment.](https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/invites/invite_760904a64b4b9dc9/accept)
:::

## What's Next?

After running the initial set of tests, you can click the run link for any of them, update the assertions, and run the scripts once more. This flow enables complete a trace-based TDD flow.

![assertions](./img/tracetest-cloud-typescript-resize.gif)

## Learn More

Please visit our [examples in GitHub](https://github.com/kubeshop/tracetest/tree/main/examples) and join our [Slack Community](https://dub.sh/tracetest-community) for more info!
